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ABSTRACT 


Within the field of e-learning, numerous studies focus on the benefits of online delivery systems and tools for learners. 
Nevertheless, there is still limited understanding of why learners perform better or worse and experiments are only rarely 
the method of inquiry. This study reports on findings of a repetitive factoral experiment in an ecological setting with 151 
secondary school pupils in order to scrutinize crucial determinants for perceived and observed benefits of two delivery 
modes: face-to-face versus online instruction. For these purposes, the study incorporates the DeLone and McLean’s 
information systems success model, extended to cover perceptions of enjoyment. The findings show that pupils’ 
performance in the e-learning condition is significantly poorer than in the face-to-face condition. The results further point 
to the dominant position of perceived enjoyment as a determinant of satisfaction and e-learning preference. By examining 
system and individual antecedents of learning performance in an experimental design, we contribute to the body of 
knowledge regarding online learning effectiveness. The study’s limitations and opportunities for further study are 
discussed. 
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1. INTRODUCTION 


In the past decade a lot of research effort has been put in explaining e-learning success in terms of learning 
performance and (continued) deployment of systems by individuals and organizations. Despite the increasing 
attention, there is limited understanding of why learners, especially pupils within secondary schools, perform 
better or worse, or how their personal preference for online or offline approaches is affected. 

In this paper we describe an experimental study focused on the beneficial outcomes of a particular 
e-learning (EL) approach in a secondary school, as compared to a more traditional offline approach, and on 
assessing potential influencing variables related to the EL system and individual characteristics. The study 
was initiated by a group of Belgian secondary schools, searching for evidence about the benefits and 
drawbacks of the EL system they had been using for several years. In order to keep up with the fast pace of 
technological change and to better adapt to the learning needs of the ‘millennium learners’ (Ananiadou 
& Claro, 2009) the headmaster conveyed a sense of urgency to employ the wide range of tools available at 
the schools in order to create a more attractive and effective learning environment. A comparative 
perspective of delivery formats was preferred: face-to-face (F2F) versus online (EL). Despite existing 
publications on such comparisons (e.g. Abdous & Yoshimura, 2010; Jang et al, 2016), none of them give 
sufficient information on how to design e-learning, using the available tools in the virtual learning 
environment (VLE) Smartschool, with the purpose of improving or at least not reducing pupils’ learning 
outcomes and satisfaction. In addition, there is a shortage of relevant experiments in real-life school settings 
in studies regarding the impact of educational technology, due to organizational complexity (Grubi8i¢ et al, 
2009) or ethical issues. 

For the purpose of understanding individual differences in benefits of using an e-learning application, 
Delone and Mclean’s Information System’s success model (ISSM) offers a well-established approach, suited 
to be used to evaluate both mandatory and voluntary use of a multitude of information systems in various 
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types of organizations and contexts. Contrary to a behavioral model like the Technology Acceptance Model 
(TAM), the ISSM concentrates on the evaluation of actionable system and information characteristics 
(Wixom & Todd, 2005). The model corroborates that the system, information, and service quality of an 
information system are the key dimensions affecting users’ intentions or actual behavior and _ their 
satisfaction. In their latest model (Delone & McLean, 2003), the construct of ‘net benefits’ is proposed as a 
key dependent factor, influenced by intention to use and satisfaction (Figure 1). 

Although the ISSM has been deployed to study a plethoria of ISs (including enterprise systems, and 
e-commerce and e-government systems), some studies also used the ISSM in an e-learning context 
(e.g. Aparicio et al, 2017; Daghan & Akkoyun, 2016; Wang et al, 2007; etc.). The findings indicate the 
importance of system and information quality and learners’ satisfaction as drivers of e-learning success. 

However, most of these reports target higher education students and analyze learners' perceptions by 
means of a cross-sectional survey. In addition to a shortage of experiments in real-life school settings, we 
further observed a limited understanding of the objective and subjective nature of the net benefits related to 
EL system usage. To our knowledge, previous contributions did not scrutinize antecedents of both learners’ 
performance (objective measure) and learners’ perceived benefits (subjective measure), and_ their 
interrelationships. We therefore propose to extend the ISSM with particular components and to apply it an 
experiment in an ecological setting, using a comparative approach. 


System Quality 


Intention to Use 


Net Benefits 


Information 
Quality 


Service Quality 


Figure 1. The Information Systems Success Model (DeLone and McLean, 2003) 


In the following, we address 2 main research questions: 1) Do secondary school pupils benefit from 
online instruction when compared to face-face instruction? And 2) Which factors contribute to individual 
differences between pupils in terms of net benefits? 


2. METHOD 


2.1 Research Design 


We conducted an experiment in a mid-sized secondary school in Belgium as a first iteration in design-based 
research, ‘a methodology (...) that seeks to increase the impact, transfer and translation of education research 
into improved practice’ (Anderson and Shattuck, 2012: 16). Following Grubi8i¢ et al. (2009), we opted for a 
factoral experimental design in which two or more parallel groups and two or more cycles are created (Figure 
2). Each teacher taught the same course content using two delivery methods: online and face-to-face. So 
pupils followed two lessons, one online and one face-to-face. Immediately after instruction, their learning 
performance was assessed by means of a written test. 

We strongly advocate the use of the factoral experiment with two parallel groups and two cycles in order 
to overcome typical biases due to undesired group or topic effects. As GrubiSic’et al. (p. 595) pointed out: ‘Jf 
we find that in each cycle one and the same factor is more efficient than the other, then we can conclude that 
a certain factor is better regardless of a difference between taught domain knowledge. In the same way, if the 
same factor is more efficient in each cycle, in spite of the groups that it has been introduced in, then we can 
conclude that a certain factor is better regardless of a difference between the groups.’ A repetition of such a 
factoral approach, in several classes and grades in the underlying study, strengthens the reliability of the 
findings. Because our factoral design is reiterated 6 times with different teachers and subjects, potential 
biases due to teacher and subject effects are minimized. 
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Figure 2. The Repeated Factoral design 


2.2 Participants 


The full sample consisted of 151 pupils between 12 and 18 years old, from the 7th until the 12th grade of 
secondary education. 70% are male. 39.7% of the participants were enrolled in a general secondary education 
program, 32.5% in technical secondary education, 19% in a vocational program, and 9% in a secondary arts 
program. Within a class, 10 to 15 pupils were randomly assigned to one of the test conditions (EL or F2F). In 
the course of one week, one group received a F2F lesson and the other group received the digital equivalent. 
The next week, the order was reversed. For example, a pupil belonging to group | received a history lesson in 
EL mode. In that same week, group 2 received the F2F version of that lesson, by the same teacher. In the 
following week group 1 received F2F a subsequent history lesson, whereas group 2 received the EL 
equivalent. As a result, each pupil followed one F2F and one EL lesson. After each lesson, all pupils took the 
same closed book tests which were constructed in accordance with the regular school procedures and 
regulations. Test scores ranged from 0 to 10. In total, 24 lessons were organized, covering 5 subjects and 12 
different groups. The subjects were: Mathematics, History, Geography, French and Economics (see Figure 
2). 

Each of the 6 involved teachers prepared 2 lessons for their subject. The learning contents were part of the 
regular curriculum. Each lesson in the EL condition was developed as a digital ‘learning path’, a number of 
pre-ordered learning activities which offer ‘a road map to learners’ (De Smet et al, 2014). Each path 
contained multiple learning objects including online text materials, pictures, online exercises and hyperlinks 
to external websites and video clips. A pupil worked individually on a desktop computer, using a headphone. 
Teachers were present in the classroom, they assisted pupils in the event of a technical problem. 


2.3 Research Model 


The research model incorporats the original constructs of the ISSM, extended by using variables related to 
perceived enjoyment, and redefines the concept of ‘net benefits’ (Figure 3). In the following, we elaborate on 
the rationale behind the model and its hypotheses. 
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Information Quality 
System Quality 
Service Quality 


The observed benefits are conceived as differences in learning performance between the EL and F2F 
conditions. The latent construct of perceived benefits is defined as the ‘perceived usefulness of the EL system 
for learning, as compared to the conventional way of teaching’. In this regard, we assume that learners 
evaluate previous experiences with ICT and make a total judgment as to whether the benefits outweigh 
disadvantages (Kirkman, 2000). In line with this stance, we ascertain the value of perceived benefits and 
consider it to be a real proxy and predictor of observed (tested) benefits. The model further posits that net 
benefits are dependent on three primary beliefs: preference, satisfaction and enjoyment. 

Within a secondary school context ‘voluntary (intention to) use’ is restricted, since educational 
institutions or teachers decide on when and how to use EL tools. As a consequence, we included the concept 
of ‘preference’ as a replacement for the ISSM’s ‘intention to use’, as proposed by Bourgonjon et al (2009). 
Preference is defined as ‘the positive and preferred choice for the continued use of e-learning systems in the 
classroom’. We contend that preference for a system will affect pupils’ learning experiences positively and 
thus lead to better results, i.e. improved net benefits. Satisfaction is another principal determinant, expressed 
as ‘the feelings and attitudes that stem from aggregating a user’s efforts and benefits from using an 
information system’ (Wixom and Todd, 2005). As such, satisfaction is conceived as an aggregated attitude 
that positively influences the system’s net benefits (as proposed by DeLone and McLean, 2003). Perceived 
enjoyment is defined as ‘the extent to which using technology is perceived to be enjoyable in its own right, 
independent of any performance consequences’ (Cheng, 2011). Sufficient evidence validates perceived 
enjoyment as a significant driver of intention to use, usage and ease of use of EL systems (e.g., Cheng, 2011; 
Teo & Noyes, 2011). We argue that an increase in perceived enjoyment stimulates pupils to make use of the 
EL course materials. 

The ISSM further distinguishes between the system quality, the quality of the information that is provided 
by the system, and the quality of the service that is provided to use the system. In our context, we defined 
information quality as ‘the level of difficulty of the EL system, the multimedia components and the related 
learning activities’. System quality was, in line with the ISSM literature, conceived as ‘the degree to which 
the system is easy to manipulate and interact with’. 

Pupils were not trained beforehand; if necessary they received technical support by a teacher during the 
lesson. Thus, in this study service quality is conceived as the degree to which support was available and 
sufficient. Congruent with the ISSM, we propose system, information and service quality as determinants of 
preference and satisfaction. 

In the literature, enjoyment has been established as a direct determinant of the use of learning systems (or 
the usage intention) (cf. Saade et al, 2008). Using preference instead of (intention to) use, it is plausible to 
contend that if a pupil enjoys the sheer use of the EL environment, s/he will likely prefer its further use and 
be more satisfied with it. Consequently, we propose perceived enjoyment as a determinant of both use 
preference and satisfaction. 


Perceived Benefits 
Preference 
5 
Observed Benefits 


Satisfaction 


Figure 3. The Research Model 
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2.4 Measures and Analyses 


Several previously validated instruments were employed (cf. Wang et al, 2007). Both self-reported 
(perceived) and objectively measured (observed) gauges were used to evaluate the pupils’ learning 
performance. The observed benefits were calculated as differences in test performance between pupils in the 
EL and F2F condition, resulting in a differential score between -10 and 10. Perceptions were measured as 
latent constructs, using multiple items (see the appendix). 

Table 1 shows the factor loadings and internal consistency measures for every latent construct. All 
constructs show sufficient internal consistency with Cronbach’s alphas of higher than 0.7. The composite 
reliability and average variance extracted (AVE) are both above the expected thresholds of 0.7 and 0.5 for all 
the constructs, suggesting adequate convergent validity (Hair et al, 2006). While omitting the display of all 
item cross-loadings in the interest of brevity, we confirm that discriminant validity is established and that all 
items loaded on their constructs as expected. In the case of multiple determinants, the variance inflation 
factor was below 2.0 level, excluding collinearity issues. 


Table 1. Internal consistency of the measures 


Construct Items & their Factor Loadings Composite | Cronbach’s | AVE 
Reliabili Alpha 
Perceived Benefits PB1: 0.79; PB2: 0.86; PB3: 0.76 0.85 0.73 0.65 
Preference Pref1: 0.90; Pref2: 0.94; Pref3: 0.90; Pref4: 0.92 0.95 0.93 0.83 
Satisfaction Sat1: 0.88; Sat2: 0.72; Sat3: 0.88 0.87 0.78 0.69 
Perceived Enjoyment | PE1: 0.90; PE2: 0.94; PE3: 0.92 0.95 0.91 0.85 
Information Quality IQ1: 0.68; IQ2: 0.81; 1Q3: 0.80; 1Q4: 0.83; 1Q5:0.83 0.89 0.85 0.63 
System Quality SQ1: 0.77; SQ2: 0.67; SQ3: 0.88; SQ4: 0.73 0.83 0.72 0.55 
Service Quality SeQ1: 0.91; SeQ2: 0.96 0.93 0.86 0.87 


We tested the measurement and research model by applying a partial least square (PLS) modelling 
approach using the SmartPLS application (Ringle et al, 2005). 


3. RESULTS 


3.1 Descriptives 


On average, the EL instruction was modestly accepted (see Table 2). The information, system and service 
quality received rather positive scores between 3.30 and 3.55 (on a scale of 1 to 5); with a mean score for 
satisfaction of 3.09. Perceived benefits, perceived enjoyment, and in particular preference, however, received 
scores below the neutral point (3). Remarkably, pupils’ performance in the EL condition was significantly 
poorer compared to the F2F instruction mode (with test scores of 75.7% for the F2F versus 56.8% in the EL 
condition). It seems pupils appreciated the quality of the system and the EL materials but were not in favour 
of it and would not choose it if it were optional. 


Table 2. Global Descriptives 


Mean _ Std. D. Min Max 


Test Results F2F 7.57 2.25 0 10 
Test Results EL 5.68 2,19 0 10 
Test Results Difference -1.89 3.34 -10.0 10.00 
Perceived Benefits 2.63 0.82 1.00 5.00 
Preference 2.24 1.05 1.00 5.00 
Satisfaction 3.09 0.86 1.00 5.00 
Perceived Enjoyment 2.61 1.02 1.00 5.00 
Information Quality 3.30 0.68 1.00 5.00 
System Quality 3.55 0.66 1.00 5.00 
Service Quality 3.49 0.98 1.00 5.00 
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The most salient outcomes of a comparative analysis considering grade, subject and programme type (see 


Table 3), can be summarised as follows: 


() 


The observed benefits’ score, i.e. the differences in test results among respondents from the 


7th grade, is -4.96, which is significantly lower than the scores of the pupils in other grades. 


(2) 


The observed and perceived benefits’ score of the pupils enrolled in general secondary 


education (-3.43 and 2.29) is significantly lower than that of vocational, technical and arts pupils. 


(3) 


subjects. The ‘perceived benefits’ scores are the highest for French. 


The observed benefits’ score for maths is significantly lower than that for other course 


Table 3. Mean scores according to grade, subject and programme type 


School grade* ih gr. a gr. 10" gr. 11" gr. 12" gr. 

N 24 49 28 21 29 
Observed Benefits -4.96* -1.86 -0.25 -0.71 -1.83 
Perceived Benefits 2.31 2.65 2.90 2.43 2.74 
“ There were no students of the 8" grade 

Subjects Geography Economics French History Mathematics 
N 30 28 30 39 24 
Observed Benefits -1.97 -2 -0.03 -1.44 -4.71* 
Perceived Benefits 2.79 2.79 2.96* 2.39 2.22, 
Education Type General Vocational Arts Technical 

N 49 28 14 60 

Observed Benefits -3.43* -1.68 -0.64 -0.25 

Perceived Benefits 2.29* 2.79 2.55 2.90 


* : Significance is based on Kruskal-Wallis or 1-way Anova 


3.2 The Tested Structural Model 


The tested model is depicted in Figure 4. Enjoyment plays a key role in explaining satisfaction and use 
preference. System and service quality have no significant impact. Information quality and satisfaction are 
significantly related (B=0.20; p<0.05). The impact of enjoyment on the net benefits is fully mediated via 
satisfaction and preference. Preference has a strong impact on perceived benefits (B=0.47, p<0,001) but a 
negative impact on observed benefits (B=-0.29, p<0.05). Thus, the data suggest that pupils who are in favour 
of EL system usage are not the pupils with the best test results. Satisfaction is only weakly related to 
perceived benefits. It has no impact on the observed benefits, but a considerable effect on preference. 
A strong relationship exists between perceived and observed benefits (B=0.45; p<0.001). 


System Quality 


Perceived 
Benefits 
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Information een, SEE ENS Are 
Quality m Bee 


Satisfaction 
R7=59% 
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Benefits 


R2=12% 


Figure 4. The Tested Structural Model 
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4. DISCUSSION 


4.1 Findings 


Our experiment reveals that secondary school pupils have better test results in a face-to-face instructional 
situation. The pattern is consistent across subject, grade and educational type (despite some differences (see 
table 3)). This finding is also strengthened by rather low overall scores on the perceived benefits (M=2.63) 
and on pupils’ preference for using the system (M=2.24). Given the radical nature of the EL approach, where 
pupils had to deal with the online contents without further assistance, and the novelty of it in secondary 
education in Belgium, these results are understandable. The explanatory model, built upon the ISSM that we 
tested, helps to explain why the EL instruction mode resulted in lower scores on perceived and observed 
benefits. 

The model includes both instrumental and emotional determinants of preference and satisfaction. It is 
clear that the emotional variable, perceived enjoyment, is the most important determinant of satisfaction and 
preference, the latter being a dominant predictor of net benefits. Perceived enjoyment has a modest mean of 
2.61 and its impact on user preference and satisfaction is strong and significant. In other words, if pupils 
think they (will) enjoy working with an EL system, they really prefer its use and are satisfied with it. 

The instrumental determinants that focus on observable qualities of the deployed system, i.e. service, 
system and information quality, are not the reason for the lack of preference for the EL instruction mode. 
Only information quality impacts satisfaction significantly, indicating that a higher level of appreciation of 
the EL contents slightly increases satisfaction. 


4.2 Contributions and Limitations 


Our approach not only targets the less-studied group of secondary education learners, but also integrates a 
research model with an experimental design in an ecological setting. Moreover, it includes both perceived 
and objectively measured learning outcomes, and the construct of ‘preference’ as a substitute for ‘intention to 
use’. The latter is particularly pertinent in a context of non-voluntary usage, which is common in secondary 
education. The resulting validated model is re-usable for future research endeavors in- and outside the 
context of secondary education research, and it can be applied within an experimental design or in a mere 
survey-based investigation. 

On a methodological level, by applying the factoral approach we have been able to minimize typical 
biases due to undesired group or topic effects. The repetition of the design (in several classes and), only 
strengthens the findings. 

A first plausible limitation of the study is the fact that teachers were only allowed to intervene in the event 
of technical problems. In other words, a quite radical EL approach was introduced. A second limitation is 
related to the limited number of cycles employed within the larger framework of a design-based research 
(DBR) approach. The quest for sound, validated sets of guidelines for educational design, adapted to one’s 
organization, preferably involves multiple iterations in a longitudinal set-up (Anderson and Shattuck, 2012) 
and multiple methods (Wang et al., 2011). 


4.3 Implications 


At first glance, our findings are somewhat contradictory to the current body of research findings that stress 
the importance and beneficial contributions of e-learning. However, our results should not be regarded as 
being opposed to such a stance. In essence, we have explored the limits of EL in secondary education and our 
results primarily point to aspects and considerations that need to be addressed when introducing EL systems 
in schools. 

In particular, when following new trends in education, such as ‘flipping the classroom’ (FtC), one should 
take into account the nature of the materials and learner characteristics. In a context where pupils are used to 
conventional classroom teaching, the transition to more EL instruction is not evident, not even for millennials 
(the subjects of the underlying study). We contend that pupils today may be technology-oriented but when it 
comes to learning, they have limited proficiencies. Reading and comprehending learning materials 
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independently and on-screen, without outside guidance, seems less fruitful than conventional F2F lessons. In 
the light of FtC, where pupils are provided with EL artefacts they need to process themselves, our findings 
clearly mark a warning. Teaching should not be confined to providing learning materials and follow-up 
class-based activities, as pupils need to be scaffolded in their knowledge acquisition process as well. A 
principal consequence of this stance implies a shift of focus to frameworks such as ‘self-regulated learning’ 
(e.g., Lee and Lee, 2008), explaining learners’ cognitive strategies and control of their efforts; and/or the 
self-determination theory, a theory of motivation that taps into our natural tendencies during a 
task-achievement process (e.g. Jang et al, 2016). 


5. CONCLUSIONS 


This study contributes to EL research by assessing the antecedents of perceived and observed benefits of EL 
instruction as compared to F2F teaching, in secondary education. 151 pupils and 6 teachers from a secondary 
school participated in a repetitive factoral experimental design in an ecological setting. This design facilitated 
the comparison of instructional formats in an empirical sound way, while yielding interesting information for 
practitioners. The DeLone and McLean’s ISSM underpins the experimental approach. The model was 
extended with three factors: perceived enjoyment, and perceived and observed net benefits. 

Overall, both the observed test results and the perceived benefits are determinants for lower effectiveness 
rates of EL as compared to F2F classes. This finding was confirmed across subjects, educational levels and 
types. Our results show the importance of perceived enjoyment. However, pupils who preferred and enjoyed 
the system are not necessarily the best performing ones. This finding is very important. It indicates that a 
shift towards EL to achieve better learning results should not be taken for granted. We assume that the 
particular EL approach we opted for - in close cooperation with the school's headmaster and his team of 
teachers - requires from pupils that they master self-regulated learning skills. In short, schools should 
preferably try to find the optimal equilibrium between F2F and EL instruction, while focusing on both 
enjoyment and performance. 

Follow-up studies could further extend the proposed model with context and individual characteristics 
related to teaching and cognitive presence (Joo et al, 2011), alongside self-regulated learning (Lee & Lee, 
2008). 
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APPENDIX. QUESTIONNAIRE ITEMS 


Constructs Item 
Perceived PB1 To what extent is a learning in Smartschool better than a conventional class (given by a teacher) 
Benefits PB2 By using Smartschool, I can understand the course material 


PB3 By using Smartschool, I could study my lesson 

Satisfaction | Satl To what extent are you satisfied with the lesson in Smartschool? 

Sat2 Are you satisfied with this way of teaching? 

Sat3 Are you satisfied with the use of Smartschool within the school? 
Preference Pref! If I had to choose between a learning path in Smartschool and a conventional class, I would 
choose a learning path in Smartschool 

Pref2 I prefer to use Smartschool 

Pref3 If it was up to me, Smartschool would be used more frequently 

Pref4 In the future, I would like work more with Smartschool in the classroom 
Perceived PE1 I find the use of Smartschool enjoyable 

Enjoyment PE2 Learning via Smartschool is pleasant 

PE3 I find it pleasurable to learn via Smartschool 

System SQ1 Smartschool is easy to use 

Quality SQ2 To what extent did you experience technical problems during the class 
SQ3 Did you find it easy to work with Smartschool 

SQ4 How responsive was the system (the pages shown on the screen) 
Information | IQ1 Was the learning path (with its components) clear and understandable 
Quality 1Q2 How much do you appreciate the contents presented (text, video’s, pictures) 
1Q3 Were the presented assignments in the system clear 

1Q4 Could you understand the topic with the available information 

1Q5 The topic was well presented in the learning environment 

Service SeQ1 If I needed help from the teacher I received valuable help 

Quality SeQ2 The support I got was sufficient 
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